Generalized Mosaicing

We present an approach that significantly enhances the capabilities of traditional image mosaicing. The key observation is that as a camera moves, it senses each scene point multiple times. We rigidly attach to the camera an optical filter with spatially varying properties, so that multiple measurements are obtained for each scene point under different optical settings. Fusing the data captured in the multiple images yields an image mosaic that includes additional information about the scene. This information can come in the form of extended dynamic range, high spectral quality, or enhancements to other dimensions of imaging. We refer to this approach as generalized mosaicing. The approach was tested using a filter with spatially varying transmittance and a standard 8-bit black/white video camera, to achieve image mosaicing with dynamic range comparable to imaging with a 16-bit camera. In another experiment, we attached a spatially varying spectral filter to the same camera to obtain mosaics that represent the spectral distribution (rather than the usual RGB measurements) of each scene point. We also discuss how generalized mosaicing can be used to explore other imaging dimensions. 1 Multi-Dimensional Mosaics Image mosaicing is a very popular way to obtain a wide field of view (FOV) image of a scene. The basic idea is to capture images as a camera moves and stitch these images together to obtain a larger image. Image mosaicing has found applications in consumer photography [3, 13, 18, 24, 25, 28] as well as uses in various scientific disciplines [11, 16, 32, 33]. It addresses the fundamental problem of increasing the FOV without sacrificing spatial resolution. We show that image mosaicing can be generalized to extract much more information about the scene, given a similar amount of acquired data. We refer to this approach as generalized mosaicing. The basic observation is that a typical video sequence acquired during mosaicing has great redundancy in terms of the data it contains; as the camera moves, each scene point is observed multiple times. We wish to exploit this fact to explore additional aspects of imaging. Consider the setup shown in Fig. 1. A fixed filA spatially varying filter mo tio n A’ Camera Figure 1. Scene point A is imaged on the detector at A′ through a spatially varying filter attached to the camera. As the imaging system moves, each scene point is sensed through different portions of the filter, thus multiple measurements are obtained under different optical settings. ter with spatially varying properties is rigidly attached to the camera. Hence, as the camera moves (or simply rotates), each scene point is measured under different optical settings1. This simple optical filtering significantly reduces the redundancy in the captured video stream. In fact, the filtering embeds in the acquired data more information about each point in the mosaic FOV. Beside mounting the fixed filter, the image acquisition in generalized mosaicing is identical to traditional mosaicing. For example, if the filter has spatially varying transmittance, the scene is effectively measured with different exposures. These measurements can be combined to obtain a high dynamic range (HDR) mosaic. Alternatively, if the spectral band transmitted by the filter varies spatially, we obtain multispectral data for each scene point. If a different imaging dimension is of interest to the user, all he/she needs to do is change the optical filter. Note that in previous work, the enhancement of each imaging dimension (e.g., FOV, dynamic range, spectral resolution, polarization sensing, etc.) was considered separately from the others. In contrast, generalized mosaicing provides a single, unified framework to enhance all or some of these dimensions. The only requirements for implementing generalized mosaicing are a simple optical filter and algorithms for im1The filter is not placed right next to the lens, as this would only alter the aperture properties [8] without producing spatially varying effects in the image. camera spatially varying filter spatially varying filter